Local feature frameworks are difficult to learn in an end-to-end fashion due to the discreteness inherent to the selection and matching of sparse keypoints. We introduce DISK (DIScrete Keypoints), a novel method that overcomes these obstacles by leveraging principles from Reinforcement Learning (RL), optimizing end-to-end for a high number of correct feature matches. Our simple yet expressive probabilistic model lets us keep the training and inference regimes close, while maintaining good enough convergence properties to reliably train from scratch. Our features can be extracted very densely while remaining discriminative, challenging commonly held assumptions about what constitutes a good keypoint, as showcased in Figure 1, and deliver state-of-the-art results on three public benchmarks.

We present a novel deep architecture and a training strategy to learn a local feature pipeline from scratch, using collections of images without the need for human supervision. To do so we exploit depth and relative camera pose cues to create a virtual target that the network should achieve on one image, provided the outputs of the network for the other image. While this process is inherently non-differentiable, we show that we can optimize the network in a two-branch setup by confining it to one branch, while preserving differentiability in the other. We train our method on both indoor and outdoor datasets, with depth data from 3D sensors for the former, and depth estimates from an off-the-shelf Structure-from-Motion solution for the latter. Our models outperform the state of the art on sparse feature matching on both datasets, while running at 60+ fps for QVGA images.

Weaknesses: My main concerns about this paper are as follow: (1) The main contribution of the paper is not well discussed. It is not the first work using reinforcement learning to learn local features. But what is the most important difference between the proposed method and the others? Using policy gradient for discrete task learning is a reasonable idea. However, more theoretical supports are necessary.

The paper presents a new technique for learning feature matching using reinforcement learning. Strengths are good experimental results showing the technique is effective in several problems. Weaknesses are some concerns about positioning (motivation, related work), but these are addressed in the rebuttal. The final reviews are uniformly positive. This is a clear accept.

Local feature frameworks are difficult to learn in an end-to-end fashion due to the discreteness inherent to the selection and matching of sparse keypoints. We introduce DISK (DIScrete Keypoints), a novel method that overcomes these obstacles by leveraging principles from Reinforcement Learning (RL), optimizing end-to-end for a high number of correct feature matches. Our simple yet expressive probabilistic model lets us keep the training and inference regimes close, while maintaining good enough convergence properties to reliably train from scratch. Our features can be extracted very densely while remaining discriminative, challenging commonly held assumptions about what constitutes a good keypoint, as showcased in Figure 1, and deliver state-of-the-art results on three public benchmarks.

Authors introduce a novel method for learning local feature detector and descriptors in a single framework. Compared to previous works, this method uses pair of images (instead of pair of image patches) and employs data with estimated depth maps and camera parameters. Learning on full images is achieved with an image loss which bypasses the lack of differentiability by computing gradient for only one branch. Even though the method is novel, I have concerns regarding the clarity and correctness of the evaluation. The main strength of this work is the fact that it uses whole images for training the local feature detector.

We present a novel deep architecture and a training strategy to learn a local feature pipeline from scratch, using collections of images without the need for human supervision. To do so we exploit depth and relative camera pose cues to create a virtual target that the network should achieve on one image, provided the outputs of the network for the other image. While this process is inherently non-differentiable, we show that we can optimize the network in a two-branch setup by confining it to one branch, while preserving differentiability in the other. We train our method on both indoor and outdoor datasets, with depth data from 3D sensors for the former, and depth estimates from an off-the-shelf Structure-from-Motion solution for the latter. Our models outperform the state of the art on sparse feature matching on both datasets, while running at 60 fps for QVGA images.